Gaussian splats can capture a cloud surprisingly well, but they also capture the lighting used during training. That means a normal splat cloud is stuck with one baked lighting scenario.
I wanted to see whether I could store enough extra information on the splats to move a light afterwards. I’m using Blender with BlendSplat to modify the splat attributes and preview the clouds, but you can use any software that can modify and render Gaussian splats.
Thanks to Hyperbowl for letting me publish this research. I was interning there while developing the approach, and they allowed me to continue working on and publishing it. Hyperbowl
1. What are Gaussian splats?
Gaussian splats are a relatively new way to represent and render complex 3D scenes. A scene consists of many small, anisotropic Gaussian functions, each with a position, scale, rotation, opacity, and colour. They are usually optimised from input images and camera poses, often using a sparse point cloud as an initialisation.
Their colour can also be view-dependent, which lets them approximate effects such as reflections and refraction. They work well for thin or semi-transparent details such as trees and grass.
Their soft appearance also makes them useful for reconstructing volumetric subjects such as clouds. The problem is that their trained colour already contains the lighting from the input images.
2. Converting VDBs into static splats
First, we need a splat that represents the cloud: basically, a VDB converted into splats.
2.1 Dataset generation
For testing, I chose an example VDB from JangaFX: Cloud VDB in Use.
To create a synthetic Gaussian splat, I rendered 160 training images at 2880×1620. I also exported alpha so the trainer could mask out the black background.
To export the data from Blender, I used a custom fork of this addon.
The shader is a simple Principled Volume BSDF with the default grey colour. The environment is also grey-white. The goal is to create an evenly lit VDB, so the trained splat is not dominated by one baked light direction.
This is not perfectly neutral lighting: the diffuse environment still contributes some baked lighting, which can influence splat placement and introduce noise during relighting. I kept it because the diffuse colour also hides some issues in the final result. For this experiment, that was the most useful trade-off.
2.2 Training
For training, I used LichtFeld. It lets you drag and drop a COLMAP dataset and train a splat from it.
One important option is alpha masking. This makes LichtFeld respect the alpha from the rendered images instead of fitting the black background as well.
Final splat: 385k splats
3. Raycasting through the VDB
The direct way to relight the cloud is to go back to the original VDB. In a path tracer such as Cycles, light is attenuated as it travels through the volume. I can use the same idea here.
Instead of raymarching from every camera pixel, I raymarch from every splat towards the light source using Geometry Nodes. For each splat, I sample the original VDB step by step until the ray leaves the cloud.
The raymarch gives the density integrated along the light path for each splat. That is not brightness yet. Beer–Lambert law converts this accumulated density into the fraction of incoming light that reaches the splat:
This gives each splat a light-transmission value:
Nice! For one light direction, we can now bake how much light reaches every splat.
But the light is supposed to move. Baking the raymarch takes a few seconds per direction, so recalculating it whenever the light moves is not practical. We would need to store the density travelled by incoming light from every direction.
That would make each splat behave a bit like a light probe. Storing a full directional map for roughly 300,000 splats would be impractical, though. I needed a compact directional representation that could live on the splats.
This project is based on Valentine Kozin’s Sea of Thieves talk. It adapts the mesh-cloud idea to splats and uses spherical harmonics to store directional density information. Sea of Thieves: Tech Art (YouTube)
4. What are spherical harmonics?
Spherical harmonics are a compact way to store a signal over a sphere. In computer graphics, they are often used for soft, low-frequency light-probe baking.
You can think of them as a set of simple patterns drawn onto a sphere. To store a signal, each pattern gets a coefficient describing how much it contributes. Adding the patterns back together gives an approximation of the original signal.
Spherical Harmonics till level 2
The patterns are grouped into levels. Higher levels capture finer directional changes, but need more storage. At low levels, sharp cloud shadows become smooth and can create ringing artefacts. For reference, one band L has 2L + 1 patterns, while all bands up to L contain (L + 1)² patterns.
Increasing levels of spherical harmoncis
A texture on a sphere can also be described as a directional signal, which is exactly what we have here. Light has to travel through different densities from different directions before reaching each splat, so that directional density can be embedded in spherical harmonics.
5. Storing the directional density
Splats already have storage for spherical harmonics, so I use it to embed directional density information. This repurposes the view-dependent SH colour data for lighting data while keeping the trained base colour.
I cast rays through the original VDB from every splat in many random directions. Every ray gives the accumulated density that incoming light would travel through from that direction.
I project each density sample onto the spherical-harmonic patterns and accumulate the coefficients. The initially noisy samples converge to a smooth representation of the cloud. After enough rays, the coefficients form a compact approximation of each splat’s directional density map: similar to a 360° image, but sampled and stored one direction at a time.
At render time, I evaluate the SH data in the current light direction to recover approximate density. Beer–Lambert law then converts it into the amount of light reaching the splat. A single splat is only an approximation, but hundreds of thousands combine into a relightable cloud.
The result works, but it is much smoother than the reference splat. The low-order SH has stored the broad lighting, but not all of the small cloud detail.
6. Getting the detail back
The trained base colour already contains some of the small cloud detail that the SH missed. By increasing the contrast and multiplying it into the relighting result, I can bring some of that detail back. It is a bit of a cheat, but it produces a much closer result.
7. Limitations
Spherical harmonics lose high-frequency information
Cloud shadows can change quickly over direction, while the SH representation used here only stores broad changes. Higher SH levels improve this, but require more attributes per splat and make the bake heavier. The small rings and smooth lighting in the result are part of that compromise.
No scattering bake
The bake only approximates how much incoming light is attenuated before reaching a splat. It does not model how light scatters inside the cloud or towards the camera. The trained colour and contrast correction make the result look more cloud-like, but complex lighting effects are still an approximation.
Point lights are not fully accurate
The stored data only knows a direction, not the distance to the light. It works best when the light is outside the cloud, where the light path is close to the precomputed direction. A point light inside or very close to the cloud needs its path length taken into account, which this setup does not store.